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CONVERSATION REWIND 
RELATED APPLICATIONS 

[001] The following related applications are incorporated herein by reference: 

. U.S. Pat. App. Ser. No. 09/464,855 filed 12/16/99 (Attorney docket no. PHA 23,875), 

inventors Willem Bulthuis et al.; "HAND-EAR USER INTERFACE FOR HAND-HELD 

DEVICE"; 

. U.S. Pat. App. Ser. No. 09/780,229 FILED 2/9/01 (Attorney docket no. US018001, inventor 
Rik Sagar, "RAPID RETRIEVAL USER INTERFACE DESIGNED AROUND SMALL 
DISPLAYS AND FEW BUTTONS FOR SEARCHING LONG SORTED LISTS"; 

. U.S. Pat. App. Ser. No. 09/642,713 filed 8/21/00 (Attorney docket no. US 000213), inventor 
Leila Kaghazian, "SELECTIVE SENDING OF PORTIONS OF ELECTRONIC 
CONTENT". 

I. BACKGROUND OF THE INVENTION 

A. Field of the Invention 

[002] The invention relates to the field of recording real-time communications, especially in the 
consumer electronics field. 

B. Related Art 

[003] In the telephonic arts, it is known to record voice mail when the person being called does 
not answer the phone. The 911 services ("911" is the national number in the U.S. for emergence 
calls) also record telephone conversations as part of their service. However, recording services 
are not sufficiently flexible to accommodate all user needs. 

II. SUMMARY OF THE INVENTION 

[004] It is desirable to provide additional recording functionality for real-time communication. 
[005] Advantageously this is achieved by automatically recording real-time communication in 
response to a user initiating or accepting the real-time communication. Advantageously, the 
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stored communications can serve as a memory aid, or as an opportunity to forward copies of a 
conversation to additional individuals. For example, the user may want to store the 
communications at a home server, or the user may want to have the communications 
automatically transcribed and stored or forwarded as text files, etc. As another example, such 
5 automatic recording provides an advantage to the user of a mobile phone. Instead of taking notes 
or taking action while communicating in real-time, e.g., while driving, the user can postpone 
these activities to a more convenient moment as the recording is going to be available. The 
invention therefore records real time communications and stores them as data to enable people or 
p data processing systems to retrieve them. This is a helpful feature, e.g., to find back information 
t@ about past events addressed in the communication recorded, to determine or verify a certain 
context wherein certain events took place, etc. It has been known to record telephone 
conversations, e.g., by emergency services ("dial 911") and law officers. The invention, however, 
addresses the consumer functionality that preferably is integrated with the communication device 
such as a mobile phone, or with the communication service provided to consumers by the service 

H providers. 

m 

£1 [006] More specifically, the invention relates to a consumer electronics communication device. 
The device has a first functionality adapted to enable real-time communication via the device; 
and a second functionality to enable automatic recording, e.g., by default, of the real time 
communication in response to a user of the device initiating or accepting the real time 
20 communication. Preferably, the first functionality comprises at least one user interface 

mechanism for providing real-time audio and/or visual information to a user and to receive real- 
time audio and/or visual information from the user. Preferably, the device comprises a user input 
to enable the user to select a portion of the real-time communication for the recording. For 
example, while the user is discussing an issue via the device with another party, the user wants to 
25 store, for use later on, a specific piece of information being brought up in the discussion, e.g., a 
telephone number being mentioned, navigation directions, a street address, etc. The user then 
asks the other party to repeat that piece of information. While the user holds down a "record" key 
on the device, the repeated piece of information gets recorded. 

[007] Preferably, the device comprises at least one organizational mechanism for organizing the 
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recorded communications in a retrievable form for the user. The organization is based on, e.g., 

time of the day at which the communication got recorded, caller-ID, or another criterion. 

[008] The device preferably has an onboard storage for storing the recorded communication. 

For example, the device has a solid state memory card/module or a HDD module with a small 
5 form factor that stores a copy of the communication or conversation conducted in real time. The 

module is preferably removable and useable with, e.g., a laptop or desktop PC. 

[009] In another embodiment of the invention, the device has an output for transmitting a signal 

to a storage, the signal being representative of the real time communication for being recorded at 
O the storage. The device is connected via a wire or in a wireless fashion with the storage. For 
igj example, the device has a mobile telephone and the storage comprises a separate module that 

^ communicates with the mobile telephone using Bluetooth or another short-range communication 

SI 

\f\ protocol. The storage module can be kept in a suitcase or carried in another convenient place, so 
jjjj that the weight or shape does not interfere or hamper usage of the phone. 
?_ [010] The device preferably comprises an opt-out mechanism in order to exempt a particular 
{fi communication from being recorded. For example, the opt out mechanism comprises a prompter 
j^j that asks at least the user or another party involved in the real time communication to signal if 
p there is a desire for omitting the recording. As another example, the opt out mechanism 

comprises a distinct, respective, physically actuatable opt out selector apparatus located on an 

exterior of the device. 

20 [011] Preferably, the device comprises a last-communication rewind mechanism for replay of at 
least a part, e.g., the last 5 minutes, of the communication recorded most recently. For example, 
the last-communication rewind mechanism comprises a distinct, respective, physically actuatable 
selector mechanism located on an exterior of the device to retrieve this part. As another example, 
the device is voice -controllable and the rewind mechanism is activated upon a specific voice 

25 command, in order to enable hands-free operation. The storage of the last N minutes is achieved, 
e.g., through a circular buffer that starts overwriting itself when full. 

[012] An embodiment of the device comprises a thumbwheel for scrolling through recorded 
communications. A thumbwheel enables easy operation while the device is being held in one 
hand. 
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[013] The invention is particularly, but not exclusively, relevant to a device as discussed above 
that comprises a mobile phone. 

[014] An embodiment of the invention is represented by a method for offering a service to 
subscribers. The method comprises enabling automatic, or default, recording of at least part of a 
5 real-time communication initiated or accepted by at least one subscriber. The method comprises 
enabling playback of subscriber-selected ones of the recorded real-time communications, 
responsive to at least one subscriber request. In this manner, the recording and storage can be 
delegated to a network, e.g., a dedicated server on a communication network. As a result, the 
g communication device can be kept lean, and legacy devices can make use of the service. The 
m network enables organizing the recorded communications for retrieval and playback by at least 
fy one subscriber. For example, a first subscriber would like to have his/her recordings organized 
according to time: the recordings most recently made are presented at the top of a list. Another 
p subscriber likes to organize the recordings according to caller-ID. Yet another subscriber likes to 

ru 

a store the recordings in an audio format, whereas still another subscriber wants to have the 

ljgj recordings stored in a text format, created through automatic transcription from audio using 

y. speech recognition software. A specific subscriber may want to have the option to be able to 
□ it...,. 

p organize or represent the recordings in a vanety of ways, examples of which are discussed above, 
M 1 dependent on his/her needs of the moment. A proper user interface (e.g., graphical or voice 

controlled) then enables the subscriber to select the mode of organization or representation. As 
20 the recording, processing and storage is delegated to a dedicated server, quality of the recording, 

processing and organizing can be optimized and/or customized. These activities need riot be 

compromised either, as the dedicated server is professionally maintained by the service provider 

using state-of-the art software to implement the service. 

[015] The service preferably comprises offering subscribers and/or their interlocutors an opt-out 
, 25 feature so that given ones of their communications will not be recorded. The service preferably 
enables the subscriber to select a portion of the real time communication for recording. For 
example, pushing a button on the device during the communication automatically activates the 
recording of the succeeding portion of the communication, until the button is released. 
[016] The service preferably comprises offering subscribers a last-communication rewind 
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feature for retrieval of a copy of at least part of the real time communication recorded most 
recently. 

[017] An implementation of the invention uses a module with a storage for coupling with the 
communication device for enabling automatic recording of the real time communication in 
5 response to a user of the device initiating or accepting the real time communication. The module 
is coupled galvanically, electrically or in a wireless manner with the device. Preferably, the 
module is controllable via a user-interface of the device. For example, a certain key or pattern of 
keys is reserved for user-interaction with the module to implement, e.g., scrolling through the 
communication stored on the module, or replaying the communication most recently recorded, or 
<£} to activate the recording during a real time communication upon a user input. The module 
jU preferably, comprises at least one organizational mechanism for organizing the recorded 
N communications in a retrievable form for the user. The module preferably comprises an opt-out 
y3 mechanism adapted to exempt a particular communication from recording. 

[018] The invention also relates to a storage medium readable by a data processing device. The 
0 medium carries software adapted to perform operations comprising: detecting initiation or 
^ acceptance of a live communication by a user of a consumer electronics communication device; 
and recording the live communication, preferably under user-determined or selectable conditions. 
The software may also enable to forward the recording to a remote destination, e.g., determined 
by default or specified by the user or the communication system. Further objects and advantages 
20 will be apparent in the following. 

III. BRIEF DESCRIPTION OF THE DRAWING 

[019] The invention will now be described by way of non-limiting example with reference to 
the following drawings, wherein: 
25 [020] Fig.l is a schematic diagram of a device in accordance with the invention. 
[021] Fig.2 shows a network embodiment. 

[022] Fig.3 is a flowchart of certain operations in accordance with the invention. 
[023] Fig.4 is a block diagram of an implementation of the invention in a cell phone. 
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IV. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 
[024] The term "communication" shall be used herein to include telephone calls, whether audio, 
video, or multimedia in nature, and whether or not occurring via conventional telephonic devices 
or via the Internet or another data network. The term "conversation" shall be synonymous with 
5 communication and shall not exclude non-audio media. 

[025] Fig.l is a schematic diagram of a consumer electronics device 101 in accordance with the 
invention. The particular consumer electronics device pictured is a mobile phone. However, the 
invention is equally applicable to other types of phones, including videophones, or other real- 
O time communications devices, which might include personal computers connected to the 
iqq Internet. Visible in the diagram are a display 102, a keypad 103 and additional functionality 
W specification apparatus 104. The functionality specification apparatus 104 shown includes an opt- 
f/j out-of-recording button 105, a last-conversation-rewind button 106, and a thumb wheel 107, 
£ which may be in accordance with U.S. Pat. App. Ser. No. 09/780,229 referred to above. These 
functionality specification devices may be fixed function or may be programmable. Other 

O 

i|g options for specification of functionality, such as touch screens or pointer devices, are well 
H known to those of ordinary skill in the art and may be readily substituted for the apparatus 
p shown. 

[026] Fig.2 shows a network embodiment of the invention. A plurality of consumer electronics 
devices 201 are networked to a central server 204. The server 204 includes a memory 206 and a 

20 processor 205. Each consumer electronics device 201 also includes a processor 202 and a 

memory 203. The consumer electronics devices 201 include real time communication devices, 
such as the cell phone of Fig.l, any other type of phone, a personal computer, or a television with 
a set top box. The network may include a telephone network, a LAN, the Internet, or any other 
type of analog or digital network. The connections 207 may be wired or wireless. 

25 [027] Fig.3 is a flow chart showing operation of the invention. These operations may occur in a 
local processor 202, or in a central processor 205 for a network, or they may be distributed 
between the local processor and the network server. 

[028] At 301 it is tested whether a communication is started. Typically, communication is 
started when the user answers the phone or when the user makes a phone call. It may or may not 



6 



US 018119 



be desirable to begin recording as soon as the user starts using the phone. A call can be detected 
in response to hearing tones transmitted from the phone. It may be preferable to wait until an 
actual two-way communication is established, either when a remote user has picked up the phone 
or when the voice mail of the remote user has answered. For instance, if the user initiates the call, 
recording could start when a voice is detected on the other end. 

[029] As used herein, the term "remote user" does not necessarily mean that the other user is at 
a great distance. The remote user might be just down the hall from the first user. The term 
"remote user" means a user other than the one who is actually using the device that is originating 
the subject recording. The remote user need not have any control over the same device or have a 
compatible device. 

[030] At 302 it is tested whether anyone has opted out of recording. Opting out may be in the 
form of pressing of the button 105 by the user. This button is, for example, of the sort that needs 
to be pushed for each call, or it may be of the sort that once pushed stays down until it is pushed 
again. Or, there may be a programmable opt out option in the local communication device or in 
the network. 

[031] In addition, laws in some countries may require that the remote user be given the 
opportunity to opt out of recording. This can be achieved, e.g., by automatically prompting the 
remote user to press some key on his or her keypad prior to beginning speaking if recording is 
not desired. 

[032] If opt out is selected, then the communication is not selected per 304, and control should 
be returned to box 301. 

[033] If opt out is not selected, the communication is recorded and stored in a usable form at 
303. Storing may involve deleting extraneous information such as dial tone and numeric tones. 
Other extraneous information may be deleted as well. Information may be added to facilitate 
storing and/or retrieving the recorded call, e.g., time at which the call was made; location where 
received (location of cell phones can be tracked), etc. 

[034] The retrievable form will typically involve some form of organization of the stored data. 
If a central server 204 is used to store the communications, the organization may take any 
suitable form, including use of a keyword-oriented database. The following are examples of 
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organization strategies that might be employed by the designer: 
order of receipt, e.g. via sequential memory access; 
time and date of receipt, e.g. via direct memory access; 

some identification of the remote user who participated in the communication, such as 
5 phone number; or 

folders given user-selected names, with communications being further organized 
within the folders. The user may have communications first buffered in a cache from 
which the user can store the recording in the appropriate folder, e.g. manually or 
_^ through another user input, in the appropriate folder. Fore example, caller-ID can be 

M used to automatically allocate the recording to the proper folder, 

fit The above list only gives a few examples of the many strategies for organizing communications 
that might be employed by a skilled system and/or user interface designer. 

m 

%Q [035] Commonly, various database management techniques will need to be used. For instance, 

ry 

it may be desirable to delete all communications that are older than a certain number of days; or 

Q there may be some numeric limit on the number of communications that may be stored. 
S3 

lj, [036] The designer may choose to store a small portion of the communications, perhaps the 
2 most recent one or two, in the local memory 203, while storing the bulk of the communications 
Mi in a memory 206 associated with a network server. The thumb wheel 107 may advantageously be 

included to facilitate ease of scrolling through stored phone calls, e.g. as specified in U.S. Pat. 
20 App. Ser. No. 09/464,855 mentioned above. 

[037] The organization of the communications may, advantageously, offer a number of 
features. For instance, a "last conversation rewind" feature may be implemented. Such a feature 
may be activatable, for instance, by button 106, and would allow a user to replay the 
conversation most recently recorded. Such replay would allow taking notes or clarifying points 
* 25 that the user did not understand. The system preferably offers the following additional 
capabilities: 

the communication may be stored in digital format, e.g. wav format; mp3; etc. 
• playback controls such as: speed, volume, pause, rewind, fast forward, and so forth, 
forwarding conversations or transcoded versions thereof to other users, e.g., as voice 
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mail or as an attachment to e-mail, etc.; 

• transferring recordings to some additional medium; and 

• converting stored conversations to text via speech recognition, and forwarding such 
text via fax or e-mail to a remote user. 

[038] Operation of the server 206 may be by a commercial service, for instance by the wireless 
service provider in the case of mobile phones. The service provider may charge fees for 
recording, storing and retrieving communications. 

[039] The user of the device is preferably enabled to have a call recorded that s/he initiates as 
well as an incoming call. In one embodiment of the invention, recording starts as soon as dial 
tones or "ringing" tones are detected. Some countries use different tones for different call status, 
but the periodicity and frequency spectrum is sufficient to recognize the tones as such. For calls 
received, starting recording is not a major issue, since the recording can begin when the user 
answers the phone. For calls initiated at the device, optionally, in a post-processing step (after 
completion of the call and therefore after recording), the dial tones get deleted under software 
control to compact the recording". In another embodiment, only a single one of the busy tones or 
of the calling-in tones gets recorded using a software procedure that, in a repetitive tone 
sequence, overwrites a previous similar tone with the current one in the recording memory. In yet 
another embodiment, the abrupt break of the dial tone sequence (when the other party picks up 
the phone) may have an acoustic fingerprint that serves as a signal to start the recording. 
[040] The invention is especially relevant to users of mobile communication devices such as 
cell phones. Typically, when on the move, the user does not have the opportunity to both juggle 
the phone and write down some piece of information that he other party is supplying via the 
phone in real-time. Similarly, when making or receiving a call while driving, the user has to keep 
his/her eyes on the traffic and has to remain aware of the road conditions. It is therefore 
advantageous for the user to be able to replay the conversation if and when needed. 
[041] The recording can, but need not, be made at the device itself. For example, a cell phone 
can be equipped with a recording component comprising, e.g., a flash memory card or a hard 
disk drive unit (HDD) with a small form-factor. The recording component can be accommodated 
in the cell phone or can be made a separate module that is galvanically or wirelessly coupled 
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with the communication device. In the latter case, the module preferably uses a suitable short- 
range communication protocol, e.g., Bluetooth. The recording components as a separate module 
can be an after-market add-on. Play -out of the recording can be through the loudspeaker of the 
cell phone, of a laptop PC or through another loudspeaker, e.g. the stereo system of the user's car 
5 using a suitable connection. Alternatively, the module can be equipped with its own loudspeaker. 
[042] The recording can also be (part of) a service on a communication network. For example, 
the user can lease storage space on the network for buffering of the recordings, e.g., as is 
similarly being done for voice mails. An application server processes the communication 
Q information and converts it to a digital format suitable for a sound file within this context and for 
y| the expected quality, e.g., using the wav format. Selecting a dedicated telephone number or other 

^ unique identifier, e.g. a URL, gives the user access to his/her recordings. Alternatively to the 
n! 

yj "last conversation rewind button" described above, the communication device or recording 
J*! component may have other dedicated buttons or user input interfaces to initiate the retrieval of 
one or more of the recordings from the network in a user-friendly manner. Preferably, user- 
access to stored recordings is through an authentication process, e.g., using a password or other 
unique identifier of the user and/or of the communication device. In case the recordings are 
stored on the network, the communication device may serve as a controller to have these 
recordings made and/or retrieved from the network. Alternatively, or in addition, the recordings, 
either as audio data or transcribed text data, are retrievable from the network from another 
20 device, e.g., laptop or desktop PC with the proper connection hardware and software. 

[043] The recordings are preferably stored in digital form, possibly after sampling an analog 
signal. In this manner, compression techniques can be used to save memory space. Digital 
compression also saves bandwidth when forwarding the recording, e.g. as an attachment to an e- 
mail or as a stand-alone compressed sound file for use among audio-based communication 
25 devices. 

[044] Caller-ID or other profiling information (e.g. time of the day, geographic location, day of 
the week, etc.) may be used to select for which communications to skip the recording or to 
explicitly carry out the recording. 

[045] Fig.4 is a block diagram of a cell phone 400 according to the invention. Cell phone 400 
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comprises a microphone 402 and a loudspeaker 404. The analog signal from microphone 402 
gets sampled and converted to a digital format in A/D 406. The digitized speech is then supplied 
to a speech codec 408. As known, the digitized speech gets coded for transmission via antenna 
410, preferably in terms of parameters descriptive of the speech in order to minimize bandwidth 
requirements of the transmission. The parameters transmitted enable reconstruction of the speech 
at the receiving end. Similarly, the signal received at antenna 410 describes the speech input 
from the remote party in terms of parameters that get processed by codec 408, converted to an 
analog signal by D/A 412 for play-out via loudspeaker 404. This type of processing is known in 
the art of digital cell phones. Cell phone 400 comprises a storage 414 that stores the parameters 
)3 representative of the speech input by the user via microphone 402, and the parameters 
ry representative of the speech signal received from the remote user and rendered by loudspeaker 
N 404 in normal operational use of phone 400. Now, for a rewind of the conversation recorded, the 
parameters stored in storage 414 get supplied to codec 408. Codec 408 then reconstructs the 
digitized conversation and has it rendered by loudspeaker 404. A user interface (UI) 416 is 
provided for user control of the rewind. 
2 [046] The data stored in storage 414 thus represents a conversation recorded. In a simple 

9 implementation of the invention, storage 414 comprises a FIFO and stores an amount of data 

O 

|=* equivalent to, e.g., one minute or a few minutes of conversation before a stored part gets 

overwritten. This enables or facilitates an instant rewind. This data can be sent, or otherwise be 

20 made available, to rendering software emulating or representing codec 408. For example, the 
user could copy or download the content of storage 414 onto a PDA, a set top box or a laptop, 
and have it rendered locally using codec software combined with speech-to-text conversion 
software, commercially available. The user could also have the data per recorded conversation 
stored at the laptop or PDA and transfer it as a file, e.g., as an email attachment. 

25 [047] In summary, the means of processing of the speech is already available at the digital cell 
phone, and little overhead is needed to implement the recording, automatic or user-selective, and 
the user-control (opt-out; record portion of current conversation) according to the invention. 
[048] The above invention addresses the recording of real time communications. Such 
communications are volatile, and recording has its use and advantages as explained above. This 
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concept can be expanded to cover the functionality of a Personal Logbook. The user carries a 
capturing device (e.g., a microphone, a solid state camera) that captures audible and/or visual 
impressions and/or context information for being stored. The information is preferably stored 
with time stamps and/or location stamps. In the latter case, a GPS system or another location- 
5 finding technology (e.g., beacons) provides the data for the geographic location of the user at the 
time of the recording that particular piece of information. The collection of recordings forms a 
personal log or diary /journal. Capturing can be continuous, event-triggered (e.g., heart-beat of the 
user, detection of a specific sound, coming within range of a specific object, etc.), user- 

_ controllable (voice activated, push-button) or controllable by another party. In the latter case, a 

Q 

da child can be given such a capturing device, integrated within a piece of clothing or bicycle. The 
Sj device can be remotely activated by the parent at any time. The device sends the captured data to 
H 4 a remote server or stores it locally for download and post-processing later on. 

[049] Another example of volatile information exchange is Instant Messaging. Applying again 
the concept of the current invention, the string of messages, displayed in an Instant Messaging 

9 

61 application or Chat Room application, gets copied to a non-volatile memory (e.g., HDD or solid 
£T state memory) for retrieval later on, e.g., automatically or upon specific user-input. In an 

E 

O automatic mode, the user may have specified in advance the size of memory to be allocated to 

O 

y. the recording, similarly to the last 5 -minute recording in the examples described above for the 
CE communication device. Currently, Instant Messaging text is available in the frame buffer of 

20 the PC for scrolling purposes, but there is no more permanent storage feature available to review 
the communication off-line. 

[050] From reading the present disclosure, other modifications will be apparent to persons 
skilled in the art. Such modifications may involve other features which are already known in the 
design, manufacture and use of devices for recording and retrieving real time communications 
25 and which may be used instead of or in addition to features already described herein. Although 
claims have been formulated in this application to particular combinations of features, it should 
be understood that the scope of the disclosure of the present application also includes any novel 
feature or novel combination of features disclosed herein either explicitly or implicitly or any 
generalization thereof, whether or not it mitigates any or all of the same technical problems as 
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does the present invention. The applicants hereby give notice that new claims may be formulated 
to such features during the prosecution of the present application or any further application 
derived therefrom. 

[051] The word "comprising", "comprise", or "comprises" as used herein should not be viewed 
as excluding additional elements. The singular article "a" or "an" as used herein should not be 
viewed as excluding a plurality of elements. 



